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ABSTRACT 



The authors present a framework depicting an 
evaluation-decision sequence, describe each of the evaluation stages, 
and note the relationships between evaluation information and 
subsequent decisions. In Chart I, the four major decision areas and 
the five kinds of evaluation which provide information for decisions 
in each of these areas are identified. It is hoped that this model 
for educational evaluation points out the importance of evaluation 
information in the process of improving educational programs and 
helping to ensure their success. The authors emphasize that 
evaluation should begin when an educational agency first looks 
critically at the potential of the educational process in its 
particular system and not after all attempts at improvement have been 
concluded. { Authors/AMM) 
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This paper discusses a basic model for the evaluation of educational 
systems and instructional programs. A definition of evaluation is pre- 
sented and a discussion of this definition will lead to the description 
of a decision oriented model for educational evaluation. Finally, the 
application of this framework for evaluation is applied to TESOL programs. 

Definition of Evaluation 

Most previous attempts to define evaluation have characteristically 
represented it either as a system of measurement and testing or the for- 
mulation of statements of congruence between performance and objectives, 
or of providing professional judgments. Since none of these definitions 
adequately provides the necessary information or includes the broad range 
of activities comprising evaluation, the Center for the Study of Evalua- 
tion, UCLA, has attempted to provide a more meaningful definition of eval- 
uation by focusing on the necessity for meeting the information needs of 
decision-makers. The Center stresses the need for evaluation to consider 
not only the ultimate decision-making functions to be served but also the 
nature of the specific problem or situation (e.g., instructional program) 
under analysis. Within such a frame of reference, the Center prefers the 
following definition: 

Evaluation is the process of ascertaining the decision areas of 
concern * selecting appropriate information , and collecting and 
analyzing information in order to report summary data useful to 
decision-makers in selecting among alternatives. 



Decision Areas $ Evaluation Requirements 



This general view of evaluation has led the Center to a consideration 
of the total instructional improvement process in order to identify the 
normal sequence of educational decisions that might occur. The identifi- 
cation of the educational decision sequence provides insight concerning 
the kinds of judgments that decision-makers must make and their concomitant 
information needs. In Chart 1, which follows, four major decision areas 
are identified and the five kinds of evaluation which provide information 
for decisions m these areas. The decision areas are concerned with: (1) 

selection of the appropriate problem or objective to be served, (2) selec- 
tion and design of the program to be introduced which best fulfills the 
objective, (3) modification of the program m terms of field conditions, 
and (4) certification of the appropriateness of the program for introduc- 
tion elsewhere. The five evaluation areas listed are intended to provide 
information related to these decisions . In the succeeding paragraphs , each 
of these areas is examined. 

Needs Assessment Evaluation (E^j 

It is generally a reasonable assumption that the decision-maker may 
be forced to make choices related to school or program priorities. For 
example, a decision-maker, given the scarcity of resources within his 
current situation, might want to make a decision about the problem area 
most in need of attention. Tins problem selection decision , m large part, 



is based on a needs assessment evaluation . Needs assessment attempts to 



Chart 1 



Decision Areas and Evaluation Requirements 



Evaluation 


Decision Area 


Needs Assessment (E^) 


Problem Selection (D^) 


Program Analysis (E2) 


Program Selection (D^) 


Implementation Analysis (E, .,) 
(Intervention) 


Program Modification (D^) 


Process Analysis (E» ? ) 
(Intervention) 




Outcome Analysis (E.) 
(Non- Intervention) 




Program Certification (D^) 
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examine the gap between goals and the existing state of affairs. The 
evaluative problem, therefore, is essentially one of assessing the needs 
of students, of the community, and of society m relation to the current 
status or accomplishments of the system. 

Once he has been provided with this information, the decision- 
maker concerned with the instructional improvement process is able to 
select from among the alternatives those problem areas (objective 
dimensions) which need attention or modification. 

Program Analyses Evaluation (E-) 

A second decision area of concern deals with program selection. 
Having made a decision about the specific problem area (or objective) 
in which his system is deficient, the decision-maker must choose be- 
tween the existing program which is directed towards the achievement of 
that objective and various alternative programs that might be introduced 
to meet that objective. These choices, referred to as program selection 
decisions , are in large part based upon information provided m a program 
analysis evaluation . In this evaluation stage, generally, the evaluator 
is asked to provide information on the possible impact of the introduc- 
tion of several alternative programs. His responsibility is to provide 
to the decision-maker, prior to a program’s inception, information con- 
cerning the potential success of these programs. Thus, while it may ap- 
pear that the evaluator’s task is merely speculative, he must attempt, at 
the very least 5 to develop procedures for systematizing speculation. For 
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instance, review procedures for examining the quality of the program 
materials might be developed, or simulations and games might be util- 
ized to examine, in microcosm, the potential impact of the various 
alternatives . 

A program analysis evaluation is objective -oriented m that the 
alternative programs, including those which are developed during the 
evaluation, are being analyzed relative to their potential accomplish- 
ment of the desired objective (s) . Moreover, potential attainment of 
other objectives (desired or undesired) are also a basis for consideration 
of programs. 

When he has been provided with such information, the decision-maker 
must then make a judgment as to which program offeis the greatest prob- 
ability of success within his system. This decision, however, is not 
entirely dictated by the data which the evaluator has provided. Even 
though the evaluator should attempt, within his study, to be as cogni- 
zant as possible of the political and contextual variables which bound or 
restrict the nature of the ultimate decision, it is unlikely that he can 
become aware of them all. The program actually selected, thus, may differ 
somewhat from the one which looks best M on paper." 

Implementation Analyses (E^ and Process Analyses (E^ 2) Evaluations 

Once he has made the decision to introduce a specific program, the 
decision-maker, throughout the course of the introduction of that program, 
must be in a position to make program modifications as demanded by the 
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situation. In order for the decision-maker to be able to make these pro- 
gram modification decisions , he must have evaluation information of vari- 
ous types. First, there is a need for information concerning the extent 
to ydiich the program has been implemented in the manner presribed during 
the program selection and to the group for whom it was intended. This 
information is referred to as implementation analysis evaluation . Secondly, 
there is a need for the decision-maker to have information on the impact 
of the program on the educational process. This process analysis evalua- 
tion is intended to provide the decision-maker with information concerning 
the progress being achieved towards fulfilling the intended objective (s) 
as well as information on any unanticipated outcomes which might have been 
observed. 

In an implementation analysis evaluation the emphasis is upon pro- 
viding the decision-maker with information on the manner in which the 
program has been implemented. One relevant question is the degree to 
which that program, as described in the program selection decision, has 
been introduced. That is, has the program been introduced and is it 
operating in the pedagogical manner in which it was intended to be used? 
Furthermore, since the program selection decision was based on various 
assumptions about the nature of the student population to be served, if 
these assumptions were incorrect or are no longer appropriate, then this 
information, as a part of an implementation analysis evaluation, should 
be meaningful to the decision-maker in determining whether he should allow 



the program to continue. Thus, the nature of the student population is 
another source of data in an implementation analysis evaluation. 

In terms of these considerations, it is apparent that the evaluator, 
in this evaluation stage, does not consider the extent to which the objec- 
tives have been achieved. 

In a process analysis evaluation, the evaluator has been asked to 
provide data on how the program is functioning. For example, in terms of 
short range objectives, such as the intended objectives to be achieved at 
the end of a specific unit of study, what has been the performance of a 
given student group? Are there any observable unanticipated outcomes which, 
though not a part of the original objectives for the program, nevertheless 
ought to be considered as info?rmation potentially valuable to a decision- 
maker in making decisions about the program? 

A process analysis evaluation is objectives -oriented. That is, the 
principal data obtained by the evaluator are progress towards the achieve- 
ment of objective (s) and unanticipated outcomes. 

The decisions related to implementation analysis evaluation and pro- 
cess analysis evaluation are likely to be in the form of program modifica- 
tions which are made during the course of the program rather than decisions 
made at the conclusion of the program. The evaluative functions identified 
in the two types of evaluation leading to program modification decisions 
are designed to be interventionist in nature. That is, during these stages 
the evaluator is concerned with providing information regarding the modifi- 
cation and improvement of programs during the process of their introduction. 
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This function is directly opposed to the more passive role that might be 
assumed by a research- observer who, because of his desire to draw valid 
generalizable conclusions, is careful not to intervene in the process. 

Outcome Analyses Evaluation (E^) 

After a program has been introduced and properly implemented into the 
system and has been modified in terms of whatever difficulties were ob- 
served, one may wish to make decisions about the program m its entirety 
and its generalizability to other situations. These decisions, which are 
concerned with whether the program should be eliminated, modified, retained, 
or introduced more widely, are referred to as program certification deci- 
sions . The evaluation associated with such decisions are referred to as 
outcome analysis evaluations . 

In outcome analysis evaluations the role of the evaluator is modified 
from the interventionist stance previously described. In this stage of the 
evaluation it is imperative, in order to maintain the generalizability of 
the situation, that the evaluator not be actively involved in the program, 
and that he attempt to insure that drastic program modifications are not 
being made concurrently with the outcome analysis evaluation. Many proce- 
dures of experimental research would be applicable to such situations. It 
is the Center’s view that the changing nature of educational systems as well 
as the necessity of having to deal with real world problems would make it 
extremely difficult to complete a pure outcome analysis evaluation. 
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Application of the Framework for Evaluation to ESL Programs 
The ensuing discussion will try to describe examples of evaluation 
information and subsequent decisions for each of the evaluation stages 
previously described. It has been pointed out that decisions are associated 
with each of the five evaluation areas, and that it is the responsibility of 
the evaluator to provide the decision-maker with information that will assist 
him in selecting among alternatives for that particular decision. It has 
been emphasized in this paper that evaluation must be considered as a com- 
prehensive process that applies to all areas of decision-making in education. 

At this point, perhaps, it should be mentioned that the evaluation function 
should take place whether or not a person with the title of evaluator is 

present. In many cases, the data collection, analysis, and reporting of 

* 

information appropriate to a decision might be so easy to obtain or so tied 
to the making of the decision, that the decision-maker and his staff would 
perform the evaluation themselves. 

The first question facing the decision-maker is related to making 
choices regarding priorities among the educational needs of the school com- 
munity. The decision-maker might be presented with needs assessment infor- 
mation indicating that certain educational goals considered highly important 
by the school community are being inadequately met. One of these goals may 
be the development of an adequate self concept by students whose first lan- 
guage is other than English. The decision-maker, having made a decision 
about one specific problem area, must now choose between the existing program (s) 
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directed toward the achievement of this particular goal (if any program 
does exist) and various alternative programs that might be introduced. 

On the basis of program analysis information related to advantages and dis- 
advantages of various alternative approaches, the decision-maker decides 
that a bilingual education program would best meet the identified needs 
of the children. At this point more detailed program planning would take 
place. A proposal for special funding may be one of the products at this 
time. 

In the process of introducing the new program, the decision-maker 
can use implementation analysis information to make modifications as the 
situation demands. Thus, the decision-maker may learn that children are 
entering the new program with English language skills that are above or 
below that which was expected, making certain instructional plans and 
materials inappropriate. He may also find out that certain audio-visual 
aids required by the instructional program have not arrived from the man- 
ufacturer, that the full complement of bilingual classroom aides as required 
by the program have not been hired, or that 35 children per class instead of 
25 have been assigned- -again deviating from the original program plan, etc. 
With the help of this type of information, decisions can be reached regard- 
ing the implementation of the new program. It should be noted that, in 
each case, the decision-maker may or may not decide to change the operat- 
ing program in favor of what the program plan states the program should be. 
He will be in the situation, however, of knowing the difference between 
what was said in the program proposal and what actually is being done. 

Once the program has been implemented, the evaluator is asked to pro- 
vide process analysis information related to how the program is functioning. 
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Here the decision-maker has the opportunity to make decisions that hope- 
fully will result in improvement of the program while the program is still 
in operation (e.g. before the end of the semester). Thus, it may be learned 
that teachers are having difficulty using certain instructional techniques 
and are in need of further in-service training, or that students in a cer- 
tain classroom may not be making satisfactory progress toward specific pro- 
cess instructional objectives of the program related to the goal of learning 
more about the history and culture of their native land. This is found to 
be due to a lack of understanding of the program's objectives on the part 
of the teacher. It becomes possible, therefore, for the decision-maker to 
modify the program while it is still going on and to increase the probability 
of success of the program. 

Finally, the decision-maker must make decisions regarding whether the 
program in question should be eliminated, modified, retained or recommended 
for wider use. The outcome analysis evaluation information might include 
statements regarding the progress of students in the local program towards 
various program objectives, as compared to an equivalent group of students 
in a nearby school. The particular- evaluation design used by the evaluator 
should make it possible for the decision-maker to make such statements as: 

"It was the high amount of individual attention provided by the 7:1 ratio 
of children to bilingual adults that made the difference in regard to objec- 
tive #1 and not the XYZ audio-visual aids"; "ABC curriculum materials that 
were prepared by the teachers helped the project pupils perform better in 
relation to educational objectives #5-10 than did EFG materials"; "The 
average number of "tardies" of children in the experimental program was 501 
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less than the average of the school/' Decisions can thus be made as to 
which parts of the program, if any, to Deep , which parts to modify, and 
perhaps most important, what to recommend to other schools regarding what 
works with which types of children. This kind of information is very 
helpful, of course, in developing or revising future program plans. 

Summary 

In the preceding pages a framework depicting an evaluation- decision 
sequence has; been presented. Each of the evaluation stages has been de- 
scribed and the relationships between evaluation information and subsequent 
decisions have been noted. In Chart 1, the four major decision areas were 
identified and the five kinds of evaluation which provide information for 
decisions in each of these areas. It is hoped that this model for educa- 
tional evaluation points out the importance of evaluation information in 
the process of improving educational programs and helping to ensure their 
success. It has been emphasized in this paper that evaluation should begin 
when an educational agency first looks critically at the potential of the 
educational process in its particular system and not after all attempts 
at improvement have been concluded. 



